Automated event detection and photo product creation

ABSTRACT

A computer-implemented method for automatically detecting events and creating photo-product designs based on the events in a photo-product design system includes automatically identifying an event by an event detection module based on daily numbers of captured photos over a plurality of days, automatically selecting a photo-product type by an intelligent product design creation engine in the photo-product design system, calculating a daily weight for a photo product design in the photo-product type based on the daily numbers of captured photos, automatically determining a number of product photos allocated to each day based on associated daily weight, automatically selecting product photos from the captured photos each day at the event according to the number of product photos allocated to each day, and automatically creating a photo-product design for the event using the selected product photos.

TECHNICAL FIELD

This application relates to digital imaging technologies and photoproducts, and in particular, to technologies that can automatephoto-product designs with no or minimal user input.

BACKGROUND OF THE INVENTION

With the advancement of consumer electronic devices, a vast number ofpictures are taken by mobile devices and digital cameras. Digital imagescan be incorporated into product designs for products such as photobooks, photo calendars, photo greeting cards, posters, photo decor,photo mugs, photo T-shirts, photo magnets, photo coasters, photo aprons,and so on.

A photo product can include one or more pages. Designing a photo productcan include many iterative steps such as selecting an overall style,selecting suitable images, selecting a layout for each page, assigningimages to each page, inputting text, rearranging the pages, the imagesand the text. The product creation process can be quite time consuming.Increasingly, photo products are designed using mobile devices.

There continues to be a need for a method to enable users to designphoto products in a convenient and time efficient manner.

SUMMARY OF THE INVENTION

The present application discloses system and method that cansignificantly enhance users' experiences associated with designingpersonalized photo products. Automated tools have been developed toautomatically detect events from the photos taken by user devices and toautomatically create designs of photo products.

An advantageous feature of the presently disclosed invention is that theactivities and events that a user has been participating areautomatically identified without user input. These identified events andactivities are used as triggers to create photo-product designs withoutuser initiation. The photo-product type, style and layouts areautomatically selected for the event. The disclosed method and systemcan significantly save users' time and make it much more convenient forusers to use their photos on their devices.

Another advantage of the presently disclosed invention is that disclosedmethods and systems can proactively identify possibilities for photoproducts that the users themselves may not have realized, which help theusers to preserve their memories.

In one general aspect, the present invention relates to acomputer-implemented method for automatically detecting events andcreating photo-product designs based on the events in a photo-productdesign system. The method includes automatically identifying an event byan event detection module based on daily numbers of captured photos overa plurality of days; automatically selecting a photo-product type by anintelligent product design creation engine in the photo-product designsystem; calculating a daily weight for a photo product design in thephoto-product type based on the daily numbers of captured photos;automatically determining a number of product photos allocated to eachday based on associated daily weight; automatically selecting productphotos from the captured photos each day at the event according to thenumber of product photos allocated to each day; and automaticallycreating a photo-product design for the event using the selected productphotos.

Implementations of the system may include one or more of the following.The daily weight can be determined by an associated daily number ofcaptured photos divided by a total number of captured photos in theevent. The number of product photos allocated to each day in the eventcan be determined by a product of the associated daily weight and atotal number of captured photos in the event. The computer-implementedmethod can further include automatically merging adjacent capturedphotos in a day into one or more scenes; calculating a scene weight forthe photo product design based on numbers of captured photos in the oneor more scenes; automatically determining a number of product photosallocated to each of the one or more scenes based on associated sceneweight; and automatically selecting product photos from the capturedphotos at each of the one or more scenes according to the number ofproduct photos allocated to each of the one or more scenes. The sceneweight can be determined by a number of captured photos of an associatedscene divided by a total number of captured photos in an associated dayin the event. The number of product photos allocated to each of the oneor more scenes can be determined by a product of the associated sceneweight and a total number of captured photos in the associated day. Thecomputer-implemented method can further include automatically selectinga product style for the photo-product design by the intelligent productdesign creation engine. The computer-implemented method can furtherinclude automatically selecting a product layout for the photo-productdesign by the intelligent product design creation engine. The step ofautomatically selecting product photos from the captured photos caninclude ranking the captured photos and automatically selecting theproduct photos from the captured photos based on the ranking of thecaptured photos. The step of automatically identifying an event by anevent detection module can include determining an average number ofcaptured photos per day; and identifying the event by the eventdetection module by comparing daily numbers of captured photos over theplurality of days to the average number of captured photos per day. Theevent can be identified by the event detection module when a dailynumber of captured photos is at least 50% higher than the average numberof captured photos per day. The event can include a single day. Theevent can include multiple days.

In another general aspect, the present invention relates to aphoto-product design system for automatically detecting events andcreating photo-product designs for the events. The photo-product designsystem includes an event detection module that can automaticallyidentify an event based on daily numbers of captured photos over aplurality of days; and an intelligent product design creation enginethat can automatically select a photo-product type, to calculate a dailyweight for a photo product design in the photo-product type based on thedaily numbers of captured photos, automatically determine a number ofproduct photos allocated to each day based on associated daily weight,automatically select product photos from the captured photos each day atthe event according to the number of product photos allocated to eachday, and automatically create a photo-product design for the event usingthe selected product photos

These and other aspects, their implementations and other features aredescribed in detail in the drawings, the description and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a network-based system for creatingpersonalized photo-product designs and making associated photo productsin accordance with the present invention.

FIG. 2 is a block diagram of a photo-product design system forautomatically detecting events and creating photo-product designs forthe events in accordance with some embodiments of the present invention.

FIG. 3 is a flow diagram for automatically detecting events and creatingphoto-product designs for the events in accordance with some embodimentsof the present invention.

FIG. 4 illustrates an example of the distribution of photos captured byone or more user devices over several days.

FIG. 5 illustrates statistics of photos captured during a multi-dayevent identified in the photos in FIG. 4 and daily weights forallocating photos captured in different days in the multi-day event to aphotobook design.

FIG. 6 illustrates the distribution of photos captured at differentscenes within a day in the multi-day event in FIG. 5 and scene weightsfor allocating photos from different scenes to a photobook design.

FIG. 7 illustrates the allocations and selection of photos to aphotobook design based on the distribution of photos between sceneswithin a day in the multi-day event in FIG. 6.

FIG. 8 illustrates a portion of a photobook design automatically createdfor the multi-day event.

DETAILED DESCRIPTION OF THE INVENTION

Referring to FIG. 1, a network-based imaging service system 10 includesa data center 30, one or more product fulfillment centers 40, 41, and acomputer network 80 that facilitates communications between the datacenter 30 and the product fulfillment centers 40, 41. The network-basedimaging service system 10, operated by an image service provider such asShutterfly, Inc., allows users 70, 71 to organize and share images via awired network or a wireless network 51, create photo-product designs,and order personalized photo products. The product fulfillment centers40, 41 can fulfill photo products ordered by users 70, 71.

The data center 30 includes one or more servers 32 configured tocommunicate with user devices (60, 61) operated by users 70, 71 throughthe Web or a mobile application, a data storage 34 for storing userdata, image and design data, and product information, and computerprocessor(s) 36 for rendering images and product designs, analyzing andorganizing images, and analyzing and understanding user behaviors andpreferences. The user data includes account information, discountinformation, order information, relationship, and important datesassociated with each user.

The users 70, 71 can view, edit, organize, and share images, and createdesigns and order personalized photo products using a mobile applicationor a browser by accessing the website. Images can also be uploaded fromthe mobile device 61 or the computer device 60 to the server 32 to allowthe user 70 and stored at the data center 30. The images or videosstored in the data storage 34, the computer device 60, or the mobiledevice 61 usually include groups of photos or videos taken at differentevents and occasions. If users 70, 71 are members of a family or a group(e.g. a soccer team), the images from the cameras 62, 63 and the mobiledevice 61 can be grouped together to be incorporated into a photoproduct such as a photobook, or used in a blog page for an event such asa soccer game.

The users 70, 71 can order a physical product based on the design of thephoto product, which can be manufactured by the printing and finishingfacilities 40 and 41. For fulfilling personalized image products, theproduct fulfillment center 40 includes a server 42 that receives thedesign of the photo product, one or more printers 45 for printingimages, finishing equipment 46 for operations such as cutting, folding,binding the printed image sheets, and shipping stations 48 for verifyingthe orders and shipping the orders to recipients 180 and 185. Examplesof the printers 45 include can be digital photographic printers, offsetdigital printers, digital printing presses, and inkjet printers. Thefinishing equipment 46 can perform operations for finishing a completeimage-based product other than printing, for example, cutting, folding,adding a cover to photo book, punching, stapling, gluing, binding, andenvelope printing and sealing. The shipping stations 48 may performtasks such as packaging, labeling, package weighing, and postagemetering. A recipient receives the physical product with messages fromthe users at locations 90, 95. The recipient can also receive a digitalversion of the design of the photo product over the Internet 50 and/or awireless network 51.

In the present disclosure, the term “personalized” (or “individualized”or “customized”) refers to content such as photos, text, designelements, layouts, or styles that is specific to a user, a recipient, agift product, or an occasion. A photo product can include a single pageor multiple pages. Each page can include one or more images, text, anddesign elements positioned in proportions in a particular layout.Examples of personalized photo products include photobooks, personalizedgreeting cards, photo stationeries, photographic prints, photo postersand photo banners, photo banners, photos on canvas, art prints, framedprints, duvet, photo bags, photo playing cards, photo T-shirts, photomugs, photo aprons, photo magnets, photo mouse pads, photo phone cases,tablet computer cases, photo key-chains, photo collectors, photocoasters, or other types of photo gifts or novelty items. Photobooks canbe in the forms of image albums, scrapbooks, bound photo calendars, orphoto snap books, etc.

In some embodiments, referring to FIGS. 1 and 2, a photo-product designsystem 200 is provided to save users' time and effort in creatingphoto-product designs to better preserve memories and tell stories. Thephoto-product design system 200 includes an event detection module 210and an intelligent product design creation engine 230, which can bothreside on a user device such as a mobile phone. The event detectionmodule 210 automatically identifies events that the user has attendedusing properties associated with the image and other information storedon the user device.

The photo-product design system 200 also includes a product type library224, a product style library 226, and a product layout library 228,which respectively stores the product types, the product styles, andproduct layouts for personalized photo products. Product types arenormally the types of products that can be manufactured at the printingand finishing facilities 40 and 41 operated by the online image serviceprovider or third party providers. Product styles and product layoutscan include pre-stored lists of styles and layouts, and can also includethose dynamically generated by the photo-product design system 200.

In the present disclosure, the phrase “product style” refers to thebackground design, embellishments, the color scheme, or other designthemes, characteristics, topics or elements of a photo product. Thephrase “product layout” (or page layout) specifies the number, thesizes, and the positions of images on a page, the gaps between theimages and at the border of the page. “Product layout” can also includepositions and sizes of text and other design elements.

The photo-product design system 200 can also include an image store 250,and a social database 260. The image store 250 stores user capturedphotos or stock photos managed by the online image service provider. Thesocial database 260 stores relationships (family members and friends) ofa user, and face images and face models for the family members and thefriends of the user.

The intelligent product design creation engine 230 can automaticallycreate a photo-product design for the an event identified by the eventdetection module 210. The intelligent product design creation engine 230uses information and analyses on the event and other intelligence suchas social data from the social database 260, to automatically selectmost suitable photos at different scenes and/or in different days of theevent to incorporate into the photo-product design. The intelligentproduct design creation engine 230 also selects a product type, aproduct style, and product layouts respectively from the product typelibrary 224, the product style library 226, and the product layoutlibrary 228.

The photo-product design system 200 can be formed by processors andmemory on a user device (60, 61) such as a mobile phone or a usercomputer. In some cases, part of the photo-product design system 200 canreside in a central location or a cloud system. For example, part of theproduct type library 224 may reside on the servers 32 and the datastorage 34 in the data center 30. Newly developed product type stylesmay be first updated at the central location or the cloud, then updatedto t user devices at scheduled times.

Referring next to FIGS. 2-4, images captured by user devices (e.g. 60,61) are stored in the image store 250. Image capture time is one of themost important properties associated with the images. The eventdetection module 210 can conduct analyses on the distribution of thecaptured images over time to identify significant events participated bythe users. The event detection module 210 first automatically detectspeaks in daily captured photos (step 310) relative to the average numberof photos captured each day. For instance, the number of captured photosfor an event day can be at least 50% higher, or 100% higher, than theaverage number of daily captured photos. For example, referring to FIG.4, if a user takes about 15 photos on the average each day, the dailycaptured photos in numbers of 80, 70, 90, 25, 75, 60 and 105 representlarge deviations from the daily average of 15, which indicates asignificant event(s) that the user has participated. The events can lastonly a single day or over multiple days.

In the present disclosure, the phrase “captured photo” refers to a photocaptured by one or more user devices. The phrase “product photo” refersto a photo to be incorporated into a photo product.

The event detection module 210 next automatically groups successive daysthat have their daily captured photos above the daily average (step315). For example, in FIG. 4, the successive days having theirrespective photo captures of 80, 70, 90, 25, 75, and 60 can be initiallygrouped into a single event. It should be noted that the event detectionmodule 210 can further analyze the content in the captured photos andpossibly divide the six days into two or more events.

The intelligent product design creation engine 230 selects aphoto-product type based on the number of captured photos in the eventsand other properties associated with the captured photos. (step notshown in FIG. 3) For example, the selection of a photobook asphoto-product design requires typically more than 50 product photos,which can be selected to be incorporated into a photobook. A photo gift,photo stationery, or home decor may be selected for an event havingfewer product photos. The intelligent product design creation engine 230can also automatically select a product style and product layout(s) forone or more pages for the selected photo-product type. (step not shownin FIG. 3) The product style selection may depend on the type of eventthat the event detection module 210 has identified. For example, geolocation metadata of the captured photos may indicate whether the photosare taken at popular vacation places. The time of the event may alsoindicate whether the event is related to graduation, a birthday, or ananniversary. Face and object recognition may also indicate whether theevent is related to a wedding, baby birth, birthday party. Productlayouts or page layouts are related to the number of pages, and thenumber of images assigned to each scene or each day, which arepreferably displayed on the same pages.

Next, a daily weight is automatically calculated for a photo productbased on the photos captured each day and the total number of capturedphotos in the event (step 320). For example, referring to FIGS. 4 and 5,the multi-day event includes 400 images. The intelligent product designcreation engine 230 selects a photobook to create based on the capturedphotos at the event. Daily weights can be computed based on the capturedphotos on each day versus the total number of photos (e.g. 400) at theevent. For instance, the daily weight for a day can be simply the ratioof the number of photos taken on that day to the total number of photostaken at the event. In FIG. 5, the daily weights for the multi-day eventare 20%, 17.5%, 22.5%, 6.25, 18.75%, and 15% respectively for the sizedays. For a single-day event, the daily weight for that day is 100%.

The number of product photos allocated to each day is then automaticallydetermined based on the respective daily weight within an event (step325). For example, referring to FIG. 5, given 100 product photos arechosen for the photobook to be designed, 20, 17, 23, 6, 19, and 15 arerespectively allocated to each day. The product photos allocated to eachday can be calculated by the intelligent product design creation engine230 by multiply the corresponding daily weight by the total number ofproduct photos in the photo-product design (e.g. a photobook).

Next, within each day, the event detection module 210 automaticallyseparates the captured photos in a day into one or more scenes (step330), which can be accomplished by merging adjacent captured photos in aday into a scene group (step 330). For example, each captured photo in aday can be initially set in a separate scene. The captured photos takenwithin a short time interval are compared. The adjacent captured photosthat have similar content and geo locations are merged into the samescene group. The process is iterated until all the captured photos in ascene group meet a predetermined criterion such as similar content,color scheme, and geo locations, etc. As a result, the captured photosin a day can be divided into one or more scene groups.

A scene weight for a photo product is calculated based on the capturedphotos per scene and the total number of captured photos in the day(step 335). For example, referring to FIG. 6, day 3 of the multi-dayevent has 90 captured photos, a day weight of 22.5%, and 23 productphotos allocated for that day to be included in a photobook design.Three scenes are automatically identified, respectively having 45, 20,and 25 captured photos. The scene weights are automatically calculatedby the intelligent product design creation engine 230 using the ratiosof the captured photos in the individual scenes over the total number ofcaptured photos in day 3. For example, the scene ratio for scene 1 canbe 45/90=50%. As shown in FIG. 6, the three scenes respectively havescene ratios of 27.8%, 22.2%, and 50%. The number of product photosassigned to each scene is determined based on the associated sceneweight within a day (step 340). For example, referring to FIG. 6, since23 product photos are allocated to day 3 in the photobook design, theproduct photos from each of the three scenes can be calculated by thetotal number of product photos allocated for the photobook for that day(23) multiplied by their respective scene weights. For example, thesecond scene is assigned with 5 product photos (22.2% times 23) to beadded to the photobook design. For a day that contains only a singlescene, the scene weight is 100% for that scene.

Captured photos are then automatically ranked within each scene (step345) by the intelligent product design creation engine 230. For example,scores of the captured photos can be calculated within each scene basedon predetermined criteria, which for example may depend parameters suchas image quality, significance to the user(s), redundancy or similaritybetween captured photos, etc. Captured photos with higher image qualityand social relevance (based on relationship stored in the socialdatabase 260) have higher scores and are ranked higher. Captured photosthat similar or redundant are trimmed and only one or a selected few areassigned with high score or high ranking. For example, referring to FIG.7, the 45 captured photos in scene 3 on day 3 are ranked, among which 12captured photos having higher scores are selected as product photos tobe incorporated into the photobook design.

Product photos are automatically selected for each scene by theintelligent product design creation engine 230 based on the imageranking and the number of product photos allocated to each scene (step350). The selections of product photos are repeated for all scenes andin all the days in a multi-day event. A photo-product design 800 (FIG.8) is automatically created by the intelligent product design creationengine 230 in part based on the product photos selected for the scenesduring the event (step 355). The photo-product design 800 can includeone or more pages 810 and product photos 820 selected from one or morescenes from an event. Afterwards, a user can review and edit one or morepages of the photo-product design 800 at a user interface 850 on theuser device. For example, user can select other captured photos 860.

The presently disclosed method and system can include one or more thefollowing advantages. The activities and events that a user hasparticipated are automatically identified without user input. Theseidentified events and activities are used as triggers to createphoto-product designs without user initiation. The photo-product types,the photo-product styles, and the photo-product layouts areautomatically selected for the event. The disclosed method and systemcan significantly save users' time and make it much more convenient forusers to use their photos on their devices. The disclosed method andsystem can proactively identify photo products that the users themselvesmay not have realized, which help the users to preserve their memories.

It should be noted that the above disclosed method and system can beused to detect other types of events and to create other type of photoproducts than the examples provided above. The detection of events onuser devices can be conducted in conjunction with other informationretrieved and analysis results acquired. Portions of the above disclosedoperations can be implemented by more than one user device, or at acentral network locations such as a cloud system. Moreover, the eventscan be identified based on other criteria than the examples describedabove.

It should be understood that the presently disclosed systems and methodscan be compatible with different devices or applications other than theexamples described above. For example, the disclosed method is suitablefor desktop, tablet computers, mobile phones and other types of networkconnectable computer devices. The photo products compatible with thepresent invention are not limited to the examples described above.

What is claimed is:
 1. A computer-implemented method for automaticallydetecting events and creating photo-product designs based on the events,comprising: in a photo-product design system, automatically identifyingan event by an event detection module based on daily numbers of capturedphotos over a plurality of days; automatically selecting a photo-producttype by an intelligent product design creation engine in thephoto-product design system; calculating a daily weight for a photoproduct design in the photo-product type based on the daily numbers ofcaptured photos; automatically determining a number of product photosallocated to each day based on associated daily weight; automaticallyselecting product photos from the captured photos each day at the eventaccording to the number of product photos allocated to each day; andautomatically creating a photo-product design for the event using theselected product photos.
 2. The computer-implemented method of claim 1,wherein the daily weight is determined by an associated daily number ofcaptured photos divided by a total number of captured photos in theevent.
 3. The computer-implemented method of claim 1, wherein the numberof product photos allocated to each day in the event is determined by aproduct of the associated daily weight and a total number of capturedphotos in the event.
 4. The computer-implemented method of claim 1,further comprising: automatically merging adjacent captured photos in aday into one or more scenes; calculating a scene weight for the photoproduct design based on numbers of captured photos in the one or morescenes; automatically determining a number of product photos allocatedto each of the one or more scenes based on associated scene weight; andautomatically selecting product photos from the captured photos at eachof the one or more scenes according to the number of product photosallocated to each of the one or more scenes.
 5. The computer-implementedmethod of claim 4, wherein the scene weight is determined by a number ofcaptured photos of an associated scene divided by a total number ofcaptured photos in an associated day in the event.
 6. Thecomputer-implemented method of claim 4, wherein the number of productphotos allocated to each of the one or more scenes is determined by aproduct of the associated scene weight and a total number of capturedphotos in the associated day.
 7. The computer-implemented method ofclaim 1, further comprising: automatically selecting a product style forthe photo-product design by the intelligent product design creationengine.
 8. The computer-implemented method of claim 1, furthercomprising: automatically selecting a product layout for thephoto-product design by the intelligent product design creation engine.9. The computer-implemented method of claim 1, wherein the step ofautomatically selecting product photos from the captured photoscomprises: ranking the captured photos; and automatically selecting theproduct photos from the captured photos based on the ranking of thecaptured photos.
 10. The computer-implemented method of claim 1, whereinthe step of automatically identifying an event by an event detectionmodule comprises: determining an average number of captured photos perday; and identifying the event by the event detection module bycomparing daily numbers of captured photos over the plurality of days tothe average number of captured photos per day.
 11. Thecomputer-implemented method of claim 10, wherein the event is identifiedby the event detection module when a daily number of captured photos isat least 50% higher than the average number of captured photos per day.12. The computer-implemented method of claim 1, wherein the eventincludes a single day.
 13. The computer-implemented method of claim 1,wherein the event includes multiple days
 14. A photo-product designsystem for automatically detecting events and creating photo-productdesigns for the events, comprising: an event detection module configuredto automatically identify an event based on daily numbers of capturedphotos over a plurality of days; and an intelligent product designcreation engine configured to automatically select a photo-product type,to calculate a daily weight for a photo product design in thephoto-product type based on the daily numbers of captured photos,automatically determine a number of product photos allocated to each daybased on associated daily weight, automatically select product photosfrom the captured photos each day at the event according to the numberof product photos allocated to each day, and automatically create aphoto-product design for the event using the selected product photos.15. The photo-product design system of claim 14, wherein the intelligentproduct design creation engine is configured to automatically mergeadjacent captured photos in a day into one or more scenes, to calculatea scene weight for the photo product design based on numbers of capturedphotos in the one or more scenes, automatically determine a number ofproduct photos allocated to each of the one or more scenes based onassociated scene weight, and automatically select product photos fromthe captured photos at each of the one or more scenes according to thenumber of product photos allocated to each of the one or more scenes.16. The photo-product design system of claim 14, wherein the intelligentproduct design creation engine is configured to automatically select aproduct style for the photo-product design and to select a productlayout for the photo-product design.
 17. The photo-product design systemof claim 14, wherein the intelligent product design creation engine isconfigured to automatically rank the captured photos and to select theproduct photos from the captured photos based on the ranking of thecaptured photos.
 18. The photo-product design system of claim 14,wherein the event detection module is configured to automaticallydetermine an average number of captured photos per day, and identify theevent by comparing daily numbers of captured photos over the pluralityof days to the average number of captured photos per day.
 19. Thephoto-product design system of claim 14, wherein the event includes asingle day.
 20. The photo-product design system of claim 14, wherein theevent includes multiple days.